3 results
5 - Construct Validity and Cognitive Diagnostic Assessment
-
- By Xiangdong Yang, Research Associate, Center for Educational Testing and Evaluation, University of Kansas, Susan E. Embretson, Professor of Psychology, Georgia Institute of Technology
- Edited by Jacqueline Leighton, University of Alberta, Mark Gierl, University of Alberta
-
- Book:
- Cognitive Diagnostic Assessment for Education
- Published online:
- 23 November 2009
- Print publication:
- 14 May 2007, pp 119-145
-
- Chapter
- Export citation
-
Summary
INTRODUCTION
Cognitive diagnostic assessment (CDA) is increasingly a major focus in psychological and educational measurement. Instead of inferring a general response tendency or behavior consistency of an examinee over a target domain of measurement, diagnostic assessment results provide a detailed account of the underlying cognitive basis of the examinee's performance by mining the richer information that is afforded by specific response patterns. Sophisticated measurement procedures, such as the rule-space methodology (Tatsuoka, 1995), the attribute hierarchy method (Leighton, Gierl, & Hunka, 2004), the tree-based regression approach (Sheehan, 1997a, 1997b), and the knowledge space theory (Doignon & Falmagne, 1999), as well as specially parameterized psychometric models (De La Torre & Douglas, 2004; DiBello, Stout, & Roussos, 1995; Draney, Pirolli, & Wilson, 1995; Hartz, 2002; Junker & Sijtsma, 2001; Maris, 1999), have been developed for inferring diagnostic information.
Although measurement models for diagnostic testing have become increasingly available, cognitive diagnosis must be evaluated by the same measurement criteria (e.g., construct validity) as traditional trait measures. With the goal of inferring more detailed information about an individual's skill profile, we are not just concerned about how many items have been correctly solved by an examinee. We are also concerned about the pattern of responses to items that differ in the knowledge, skills, or cognitive processes required for solution. Similar to traditional tests, empirical evidence and theoretical rationales that elaborate the underlying basis of item responses are required to support the inferences and interpretations made from diagnostic assessments.
13 - Measuring Human Intelligence with Artificial Intelligence: Adaptive Item Generation
- Edited by Robert J. Sternberg, Yale University, Connecticut, Jean E. Pretz, Yale University, Connecticut
-
- Book:
- Cognition and Intelligence
- Published online:
- 23 November 2009
- Print publication:
- 01 November 2004, pp 251-267
-
- Chapter
- Export citation
-
Summary
INTRODUCTION
Adaptive item generation may be the next innovation in intelligence testing. In adaptive item generation, the optimally informative item is developed anew for the examinee during the test. Reminiscent of computer versus person chess games, the computer generates the next item based on the previous pattern of the examinee's responses. Adaptive item generation requires the merger of two lines of research, psychometric methods for adaptive testing and a cognitive analysis of items.
Adaptive testing is the current state of the art in intelligence measurement. In adaptive testing, items are selected individually for optimal information about an examinee's ability during testing. The items are selected interactively by a computer algorithm using calibrated psychometric properties. Generally, harder items are selected if the examinee solves items, while easier ones are selected if the examinee does not solve items. Adaptive item selection leads to shorter and more reliable tests. In a sense, optimal item selection for an examinee is measurement by artificial intelligence.
Adaptive item generation is a step beyond adaptive testing. Like adaptive testing, it estimates the psychometric properties of the optimally informative items for the person. Beyond this, however, the impact of specific stimulus content on an item's psychometric properties must be known. That is, knowledge is required of how stimulus features in specific items impact the ability construct.
This paper describes a system for measuring ability in which new items are created while the person takes the test. Ability is measured online by a system of artificial intelligence.
19 - Psychometric Approaches to Understanding and Measuring Intelligence
-
- By Susan E. Embretson, University of Kansas, Lawrence, Karen M. Schmidt McCollam, University of Virginia
- Edited by Robert J. Sternberg, Yale University, Connecticut
-
- Book:
- Handbook of Intelligence
- Published online:
- 05 June 2012
- Print publication:
- 13 March 2000, pp 423-444
-
- Chapter
- Export citation
-
Summary
Since the Handbook of Human Intelligence appeared in 1982, the “psychometric approach” has changed dramatically. Traditionally, the psychometric approach was synonymous with the factor-analytic approach. Exploratory factor analysis was applied to discover the number and nature of the factors that underlie performance on cognitive tasks. Carroll's (1993) three-stratum model of intellect synthesizes the factors supported across hundreds of studies. Although the studies reported somewhat inconsistent factor patterns, Carroll found consistent support for several factors by reanalyzing their data with common methods of factor analysis.
However, the contemporary psychometric approach differs in three major ways from the traditional psychometric approach: (1) confirmatory approaches predominate over exploratory approaches, (2) structural analysis of items predominates over structural analysis of variables, and (3) item response theory (IRT) models predominate over factor analytic models. Thus, in the contemporary psychometric approach, confirmatory IRT models are applied to understand and measure individual differences. The intelligence construct is elaborated in confirmatory IRT models by comparing alternative models as explaining item responses. Some confirmatory IRT models include parameters to estimate the cognitive processing demands in items. These models permit items to be selected and banked by their cognitive demand features and provide results relevant to understanding what is measured by the items. Other confirmatory IRT models include parameters for person differences on the underlying processes, strategies, or knowledge structures. These models can define new types of individual differences. As will be elaborated below, parameters are included to measure qualitative differences in item responses such as relative success in various underlying cognitive operations, use of different strategies or knowledge structures, and modifiability of ability with intervention.